Installation tweaks by wdeconinck · Pull Request #32 · GridTools/atlas4py

wdeconinck · 2026-01-29T09:10:36Z

I have made time available this year to work on our plan to extend atlas4py and hand over maintenance.
As a first step I tried the new build-system, and noticed that I needed to tweak the installation a little bit.

Fix RPATH handling where atlas library is external
Faster compilation, removing unneeded parts
Reinstallation, e.g. when developing, no longer picks up a previously installed atlas4py-provided atlas/eckit library
Added more tests, and a few cosmetic changes
Bugfix in RectangularDomain construction
Changes to avoid using deprecated eckit functionality to be removed with eckit 2.0.0.

wdeconinck · 2026-02-09T09:44:34Z

Thank you for merging #30 @tehrengruber , @havogt .
I have rebased this on latest master; It should be ready for review.

havogt · 2026-02-12T10:10:46Z

I only did a brief walk through the code so far. One question popped up and I didn't follow the code to figure it out myself: how is the version managed? Is it taking always the version of atlas?

wdeconinck · 2026-02-12T13:42:16Z

I only did a brief walk through the code so far. One question popped up and I didn't follow the code to figure it out myself: how is the version managed? Is it taking always the version of atlas?

atlas4py version is not managed by the found atlas library, but that could be a strategy. I did not deviate from what was in place. I removed however duplicate places where the version was defined with only a single place remaining, in the pyproject.toml file.
This version is then defined as a CMake variable SKBUILD_PROJECT_VERSION_FULL by scikit-build-core. I then add that as a preprocessor definition into the _atlas4py.cpp __version__ attribute of the module. This means that the manually managed __version__ attribute is no longer needed.

There is still the extra variable ATLAS4PY_ATLAS_VERSION defined in pyproject.toml, which is only used to download atlas if atlas was not found. This could be introspected from the atlas4py version instead of course, if the format is clear.
With the use of externally installed atlas versions, the version of atlas4py and the actual atlas (commit hash, branch, or tag) used may differ, as then ATLAS4PY_ATLAS_VERSION variable is ignored. I added a CMake configuration warning however when the version is mismatched.

wdeconinck · 2026-02-13T16:04:19Z

Note, just rebased again on master.

tehrengruber · 2026-02-04T15:31:56Z

 PYBIND11_MODULE( _atlas4py, m ) {
    m.def("_initialise", atlasInitialise)
     .def("_finalise",   atlas::finalise);
    m.attr("version") = atlas::Library::instance().version();


Suggested change

m.attr("version") = atlas::Library::instance().version();

m.attr("atlas_version") = atlas::Library::instance().version();

or just removing it altogether?

Suggested change

m.attr("version") = atlas::Library::instance().version();

I suppose in time, __version__ and version/atlas_versionshould be the same. We might as well remove it then indeed.

@tehrengruber this is now done.

tehrengruber · 2026-02-04T15:44:39Z

 else()
    set(rpath_origin_install_libdir "$ORIGIN/${CMAKE_INSTALL_LIBDIR}")
 endif()
+set( CMAKE_INSTALL_RPATH "${rpath_origin_install_libdir}" ) # A semicolon-separated list specifying the RPATH to use in installed targets


Can you add a motivation for the 3 additional options here?

Consider the use case without these added lines, and where we have atlas detected and not being built.
I then have this error at runtime (for instance for macOS):

from ._atlas4py import * E ImportError: dlopen(/path/to/venv/lib/python3.13/site-packages/atlas4py/_atlas4py.cpython-313-darwin.so, 0x0002): Library not loaded: @rpath/libatlas.0.45.dylib

It is clear that the rpath of the found atlas libraries is not added. CMake provides the option

set( CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE ) # add the automatic parts to RPATH which point to dirs outside build tree

to automatically add the rpaths of libraries which are not built within the build-tree, introspected from the location of the library. This then adds the rpath to the external atlas library. We could probably guard this with if (build_atlas) to make sure no extra rpaths are inserted accidentally.

The next options probably do not play a role because it is more for a standard package with multiple libraries and executables involving running executables from within the build-tree and saving time on installation. But it is the good practice that we always set by default for all our software.

set( CMAKE_SKIP_BUILD_RPATH FALSE ) # use RPATHs for the build tree

When CMAKE_SKIP_BUILD_RPATH is set to FALSE, then the executables in the build-tree etc. are also built with RPATHS.

set( CMAKE_BUILD_WITH_INSTALL_RPATH TRUE ) # build with *relative* rpaths by default

That means that the build-tree executables have been built with the same rpaths we encoded for the install. Usually we have compile destinations within the build-tree in a similar structure as install: <build>/lib, <build>/bin, and the rpaths are relative "$ORIGIN/../lib". This saves time on relinking on installation, and is then just a copy.

We could omit these last two but it is good practice, and possibly speeds up installs.

tehrengruber

Looks good, thanks!

tehrengruber · 2026-04-09T14:30:27Z

    set(rpath_origin_install_libdir "$ORIGIN/${CMAKE_INSTALL_LIBDIR}")
 endif()
+set( CMAKE_INSTALL_RPATH "${rpath_origin_install_libdir}" ) # A semicolon-separated list specifying the RPATH to use in installed targets
+set( CMAKE_INSTALL_RPATH_USE_LINK_PATH   TRUE  ) # add the automatic parts to RPATH which point to dirs outside build tree


Suggested change

set( CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE ) # add the automatic parts to RPATH which point to dirs outside build tree

if (NOT build_atlas)

set( CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE ) # add external atlas to rpath

endif()

Thanks for the explanation. The if around the CMAKE_INSTALL_RPATH_USE_LINK_PATH makes sense to me, even if it is just for readability.

… library

…f/eckit#105)

wdeconinck · 2026-04-09T17:04:50Z

Thanks @tehrengruber I have followed your last recommendations and cleaned up the history. I can only see the "Squash and merge" option for merging. Perhaps it would be useful to keep the individual commits (e.g "Rebase and merge"). Could you take care of this?

wdeconinck mentioned this pull request Jan 29, 2026

CI issues #33

Closed

wdeconinck force-pushed the installation-tweaks branch 2 times, most recently from 597b52a to 6d74645 Compare February 4, 2026 14:44

wdeconinck mentioned this pull request Feb 5, 2026

Add python 3.13, 3.14, drop < 3.10 #30

Merged

wdeconinck force-pushed the installation-tweaks branch 4 times, most recently from 6d1d6b1 to 1c7d1cb Compare February 9, 2026 09:34

wdeconinck force-pushed the installation-tweaks branch 2 times, most recently from b5f514e to 3281313 Compare February 11, 2026 15:23

wdeconinck force-pushed the installation-tweaks branch from 3281313 to 7ef5085 Compare February 13, 2026 16:03

tehrengruber reviewed Feb 17, 2026

View reviewed changes

wdeconinck requested a review from tehrengruber March 6, 2026 07:55

tehrengruber approved these changes Apr 9, 2026

View reviewed changes

wdeconinck added 7 commits April 9, 2026 18:59

Installation tweaks, robust to reinstallations and externalised atlas…

790e3f3

… library

Fix bug in arguments of RectangularDomain

8ce2206

Fix eckit deprecation warnings related to acessing eckit::Value (ecmw…

826f832

…f/eckit#105)

Update examples/Atlas4Py-Test.ipynb

8683dc1

Improve testing

2b5e86d

Remove 'atlas4py.version' attribute

c91759f

Bump dev version to 0.41.1.dev2

08602bd

wdeconinck force-pushed the installation-tweaks branch from 7472b50 to 08602bd Compare April 9, 2026 17:01

wdeconinck merged commit dc05937 into GridTools:master Apr 10, 2026
27 checks passed

wdeconinck mentioned this pull request Apr 10, 2026

Port from pybind11 to nanobind #34

Draft

	m.attr("version") = atlas::Library::instance().version();
	m.attr("atlas_version") = atlas::Library::instance().version();

-set( CMAKE_INSTALL_RPATH_USE_LINK_PATH   TRUE  ) # add the automatic parts to RPATH which point to dirs outside build tree
+  if (NOT build_atlas)
+    set( CMAKE_INSTALL_RPATH_USE_LINK_PATH   TRUE  ) # add external atlas to rpath
+  endif()

Conversation

wdeconinck commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wdeconinck commented Feb 9, 2026

Uh oh!

havogt commented Feb 12, 2026

Uh oh!

wdeconinck commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wdeconinck commented Feb 13, 2026

Uh oh!

Uh oh!

Uh oh!

tehrengruber Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

wdeconinck Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wdeconinck Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

tehrengruber Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

wdeconinck Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tehrengruber left a comment

Choose a reason for hiding this comment

Uh oh!

tehrengruber Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

wdeconinck commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wdeconinck commented Jan 29, 2026 •

edited

Loading

wdeconinck commented Feb 12, 2026 •

edited

Loading

wdeconinck Feb 19, 2026 •

edited

Loading

wdeconinck Feb 19, 2026 •

edited

Loading

wdeconinck commented Apr 9, 2026 •

edited

Loading